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FOREWORD 


To  assist  the  developers  and  users  of  current  and  future  Command  Systems,  the  Systems 
Integration  and  Command  Control  Technical  Area  of  the  U.S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences  < ARI ) provides  directly  applicable  empirical  information  for  use  in 
design  decisions  concerned  will)  such  efforts  as  the  Army  Tactical  Data  System  (ARTADSI.  One 
of  the  principal  problems  in  military  data  processing  systems  has  been  that  computer  systems 
function  but  are  difficult  to  use  except  by  trained  operators.  A continuing  research  program  in 
ARI  is  directed  toward  easing  and  simplifying  the  man/computer  interface  and  increasing  total 
system  effectiveness;  the  facet  of  this  program  reported  here  concerns  the  input  of  information 
into  the  computer  system.  An  earlier  report  (TRN  226)  evaluated  several  techniques  for  inputting 
information  into  a semi-automated  information  processing  system.  The  present  publication 
examines  two  alternative  methods  of  formatting  free  text  information  into  computer-acceptable 
terminology  and  evaluates  a machine  aid  which  automatically  presents  information  useful  in 
completing  the  format. 


The  entire  research  effort  is  responsive  to  requirements  of  RDTE  Project  2Q162106A72I. 
"Command  Information  Processing  Systems,"  FY  1974  Work  Program,  and  to  special 
requirements  of  the  U.S.  Army  Training  and  Doctrine  Command. 


✓E . UHLANER 
'echnical  Director 


automated  aids  to  on-line  tactical  data  inputting 


BRIEF 


Requirement: 

To  evaluate  (1)  a computer-assisted  message  inputting  (CAMI)  aid  that  automatically  presents 
information  useful  for  completing  message  formats  and  12)  a checklist  method  of  formatting  free-text 
information  into  computer-acceptable  terminology,  for  on-line  inputting  of  tactical  data  into  an 
information  processing  system. 


Procedure: 

The  accuracy  and  speed  of  two  inputting  aids  were  each  compared  under  two  methods  of  formatting. 
One  aid  is  a CAMI  CRT  which  offers  information,  instructions,  and  lists  of  legal  entries  for  each  item  being 
entered;  the  other  is  a looseleaf  handbook  of  instructions,  legal  entries,  and  sample  messages.  In  one  format 
method  (full),  the  primary  CRT  displays  all  data  entry  categories  and  the  operator  fills  in  the  information 
he  has,  leaving  the  spaces  blank  where  he  has  no  data.  In  the  checklist  format  method,  the  operator  checks 
the  categories  displayed  on  the  primary  CRT  which  he  has  data  for  and  receives  a second  display  of  only 
those  items  checked  (plus  the  mandatory  entries).  Results  from  the  four  experimental  conditions  were  also 
compared  with  those  from  a fifth  condition  in  which  the  operator  used  only  the  handbook  and  a 
blank-screen  CRT.  Subjects  were  60  enlisted  men  who  were  divided  into  five  groups  and  assigned  to  the  five 
condition::  to  provide  data  for  an  analysis  of  variance.  The  messages  were  from  a tactical  scenario,  one  type 
transmitting  data  about  the  enemy  and  nine  types  transmitting  data  about  friendly  troops. 


Findings: 

No  significant  differences  in  accuracy  or  speed  could  be  attributed  to  use  of  the  CAMI  aid  or  the 
checklist  format.  The  fifth,  blank-screen  condition  required  significantly  longer  (an  average  4.6  minutes  per 
message)  than  the  comparable  full  format  or  checklist  handbook  conditions.  "Friendly"  messages  were 
completed  with  greater  accuracy  and  speed  than  "enemy”  messages,  perhaps  because  they  may  have  been 
simpler  to  format. 

An  unexpected  but  important  finding  was  that  over  80%  of  the  total  input  errors  were  types  which 
would  not  have  been  detected  by  a computer  error-checking  routine. 


Utilization  of  Findings: 

Some  types  of  input  errors  are  detectable  by  computer  error-detection  programs,  but  many  are  not.  If 
total  error  ,ates  are  separated  into  their  component  categories,  the  probable  • atio  of  detectable  to 
undetectable  errors  can  be  determined,  and  the  frequency  of  undetected  errors  in  a real-world  on-line 
situation  can  be  estimated. 

Since  computerized  error  detection  is  not  likely  to  be  a complete  solution  to  the  problem  of  accuracy, 
operator  proficiency  will  continue  to  be  of  great  importance.  The  poss'bility  might  be  considered  of 
incorporating  an  on-line  instruction  program  into  the  system  for  use  in  a tactical  situation  where  trained 
replacement  operators  may  not  be  available. 
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BACKGROUND 


Computer-Assisted  Message  Inputting 

Computer  technology  Is  experiencing  growing  pains.  Initial  emphasis 
in  system  design  was  on  solving  hardware  and  software  problems  sucn  as 
determining  necessary  computer  equipment  and  selecting  and/or  developing 
the  necessary  programs.  The  result  of  this  emphasis  is  evident  today: 
Computer  systems  function  but  are  difficult  to  use  by  anyone  except 
highly  trained  operators. 

Army  tactical  data  systems  are  presently  investigating  ways  to 
simplify  the  task  of  translating  free  text  message  information  into 
computer-acceptable  terminology.  Unless  the  automated  system  is  compar- 
able or  superior  in  ease  of  use  to  the  manual  system,  potential  gains  in 
accuracy  and  speed  of  information  flow  will  not  be  realized.  A message- 
input-output-device  (MIOD)  operator  finds  it  hard  to  appreciate  the 
benefits  of  the  automated  information  system  when  he  has  to  struggle  to 
enter  any  data.1  Therefore,  the  task  of  inputting  information  should  be 
simplified.  This  simplification  should  neutralize  a major  source  of 
user  irritation  and  thereby  highlight  the  benefits  of  the  system.  Only 
in  a non-threatening  environment  will  an  atmosphere  conducive  to  user 
acceptance  be  created.  As  stated  by  Baker  et  aL, 2 "Further  it  is 
realistic  to  expect  that  the  introduction  of  ADP  into  tactical  operations 
will  produce  new  human  performance  problems.  Solutions  to  these  problems 
therefore  become  of  paramount  importance." 

One  promising  approach  to  helping  the  operator  input  information  into 
a computer  data  base  is  to  use  the  computer's  ability  to  instruct  the 
operator  while  he  is  performing  his  task.  Observation  of  field  grade 
officers  retrieving  information  from  a computerized  data  base  in  ARl's 
simulated  tactical  operations  system  (SIMTOS)  experiments  3 suggested 


1 See  Baker,  "An  assessment  of  the  impact  of  automation  on  Seventh  Army 
users"  on  user  attitude  toward  the  introduction  of  automated  equipment 
(in  preparation). 

2Baker,  J.  D.,  D.  J.  Mace  and  J.  M.  McKendry.  The  transform  operation 
in  TOS:  Assessment  of  the  human  component.  ARI  Technical  Research 

Note  212.  August  1969.  (AD  697  716) 

3 Robins,  J.  E.,  L.  Buffardi  and  T.  G.  Ryan.  Research  on  tactical 
military  decision  making:  Application  of  a decision  prediction  concept 

in  a SIMTOS  environment.  ARI  Technical  Research  Note  246.  March  1974. 
(AD  708  812) 


that  officers  quickly  feel  at  ease  interacting  with  the  computer  when 
procedural  instructions  are  explicit.  Such  an  approach  should  facilitate 
inputting  as  well  as  retrieval  of  data.  Thus,  a principal  objective  of 
the  present  research  was  to  find  out  the  relative  improvement  in  input- 
ting performance  which  could  be  expected  if  the  instructional  capabilities 
of  the  computer  were  increased.  For  ease  of  reference,  this  approach  will 
be  called  CAMI  (computer-assisted  message  inputting). 


Consolidation 

A previous  study 4 had  concluded  that  "efforts  hould  be  directed 
toward  changing  rather  than  aiding  the  present  TOS  Tactical  Operation 
System)  transform  process."  It  pointed  out  the  redundancy  of  having  the 
action  officer  select  a paper  format,  fill  it  out,  and  hand  it  to  an 
input  operator.  The  report  recommended  that  one  man  transform  the  message 
directly  onto  the  CRT  screen.  Strub  found  experimentally  that  signifi- 
cantly fewer  errors  were  made  when  the  message  was  input  directly  on  the 
CRT  than  when  paper  formats  were  used  as  an  intermediate  step  (11 .2$ 
error  vs  14. 8$). 5 

A second  recommendation  about  changing  the  TOS  transform  process  was 
made  by  Baker  et  aL:6 

One  task  that  should  be  undertaken  is  that  of  scrutinizing 
existing  formats  to  determine  if  the  absolute  number  can  be 
reduced.  Considerable  overlap  in  content  exists  among  the 
present  formats.  Certain  formats  could  conceivably  be  com- 
bined because  of  this  communallty. 

In  early  1972,  the  Intelligence  and  Control  Systems  Group  (iNCSG)  of  the 
Combat  Developments  Comnand  (CDC)  established  a Format  Revision  Committee 
with  representatives  from  Project  MASSTER,  the  Computer  Systems  Comnand, 
Army  Tactical  Data  Systems,  and  ARI.  In  February  of  that  year,  they 
conducted  a test  at  MASSTER  to  assess  the  extent,  if  any,  to  which 
consolidation  of  message  formats  would  enhance  user  performance  without 
lessening  the  acceptability  of  message  processing  procedures  to  the  user. 


4Ringel,  S.,  J.  D.  Baker,  M.  H.  Strub  and  L.  L.  Kensinger.  Human 
factors  research  in  comnand  information  processing  systems — summary 
of  recent  studies.  ARI  Technical  Research  Report  11 58.  July  1969* 

(AD  694  547) 

8 Strub,  M.  H.  Evaluation  of  man-computer  input  techniques  for  military 
information  systems.  ARI  Technical  Research  Mote  226.  May  1971* 

(AD  730  315) 

8 Baker  et  al.,  1969,  op.  cit. 
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The  dependent  variable  was  tine.  Results  of  the  test  indicated  that  the 
use  of  consolidated  formats  produced  an  average  savings  of  49  seconds  per 
input  message  compared  to  the  time  using  unrevised  input  formats. 

In  addition,  ARI  and  CDCINCSG  discussed  aiding  the  on-line  tactical 
data  inputting  process  with  CAMI  and  strategies  for  reducing  the  number 
of  characters  required  to  input  a message.  If  such  reduction  were 
provider.,  the  savings  in  space  could  be  used  either  to  reduce  the  size 
of  the  input  device  or  add  inputting  instructions  to  the  CRT  format 
display. 


Research  Objectives 

As  a result  of  these  meetings,  ARI  conducted  an  experimental  inves- 
tigation of  (l)  a CAMI  aid  and  (2)  a method  for  reducing  the  number  of 
characters  necessary  to  input  a message.  The  CAMI  aid  consisted  of  a 
second  CRT  display  which  displayed  item  by  item  instructions  and  tables 
of  permissible  codes  (legal  entries)  and  abbreviations.7  The 
character  reduction  method  consisted  of  a checklist  display  of  the 
optional  categories  of  information  contained  within  a selected  format. 
Prior  to  inputting  a message,  the  subject  first  selected  from  this  list 
only  those  categories  called  for  by  the  data  in  that  particular  message. 
He  then  received  a format  containing  only  mandatory  entries  and  those 
entries  which  he  had  selected.  The  primary  objective  of  the  present 
experiment,  then,  was  to  evaluate  (l)  a CAMI  aid  which  automatically 
presented  (via  a second  display  device)  Information  useful  for  completing 
the  format,  and  (2)  a checklist  method  of  formatting  free-text  informa- 
tion into  computer-acceptable  terminology. 

A secondary  objective  was  to  evaluate  a method  of  inputting  with  no 
computer  assistance;  that  is,  the  subject  accomplished  the  free-text  to 
computer-entry  translation  task  given  only  a blank  screen  and  a handbook 
of  instructions.  While  this  method  minimized  the  number  of  characters 
needed  for  a display  device,  the  absence  of  a format  display  or  machine 
display  of  message-composition  information  was  expected  to  be  detri- 
mental to  performance.  However,  such  a condition  would  provide  baseline 
data  for  comparison  with  the  experimental  aids  investigated  here. 


T0S2  specifications  call  for  a split  screen  capability.  Two  screens 
were  used  in  the  present  experiment  for  ease  of  programming.  A 
minor  software  modification  would  incorporate  the  material  displayed 
on  two  screens  onto  a single  split  screen. 


- 5 - 


METHOD 


Subjects 

Sixty  enlisted  men,  each  with  a GT  score  of  110  or  above  and  normal 
or  corrected  vision,  participated.  Four  men  a day  completed  as  many 
messages  as  possible  during  a morning  session  (0800-1200)  and  an  after- 
noon session  (I3OO-I7OO). 


Equipment 

The  experiment  required  six  cathode  ray  tube  (CRT)  keyboard  displays 
and  four  IO5O  Selectric  typewriters,  located  in  the  ARI  Training  and 
Information  Systems  Facility  and  wired  to  a 32K  computer.  The  CRTs 
displayed  formats  and  composition  information,  and  the  typewriters 
simulated  teletypewriters  and  typed  out  simulated  free-text  messages 
from  the  field.  Necessary  manuals  and  handbooks  were  provided  for  those 
experimental  conditions  involving  manual  reference  to  glossary  informa- 
tion. 


Procedure 

At  the  start  of  each  session,  the  experimenter  welcomed  the  subjects 
and  briefed  them  on  the  general  purpose  of  the  experiment  in  which  they 
would  participate.  Each  subject  was  then  seated  in  a cubicle  and  requested 
to  read  three  typed  pages  of  orientation  material.  The  first  page  reneated 
the  general  purpose  of  the  experiment  (i.e.,  to  test  the  relative  ( x- 
ciency  of  several  methods  of  entering  data  into  a military  computer)  and 
stressed  the  importance  of  accuracy  over  speed.  This  page  is  reproduced 
in  the  Appendix.  The  second  page  was  a situation  report  describing 
scenario  background  events.  According  to  the  scenario  prepared  for  the 
experiment,  each  subject  assumed  the  role  of  MIOD  operator  in  the  HHC  of 
the  1st  Battalion,  2d  Armor,  a unit  under  the  1st  Brigade  52d  Infantry 
Division  (Mechanized).  The  subject  was  told  that  the  messages  which  he 
would  be  processing  derive  from  the  following  background  events: 

"In  the  early  morning  hours,  your  unit  was  attacked  by 
strong  elements  of  the  Circle  Trigon  Government's  2d 
Tank  Army.  Though  surprised  and  outnumbered,  your  unit 
recovered  quickly,  fought  a successful  delaying  action, 
and  finally  withdrew  in  good  order,  sustaining  minimal 
losses.  Morale  remains  excellent." 

The  third  page  contained  general  instructions  for  inputting  a message. 

After  the  suoject  had  read  these  pages,  the  experimenter  showed  him 
how  to  input  the  message  on-line  using  a practice  message.  After  com- 
pleting one  practice  message  with  the  subject  and  answering  any  procedural 
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questions,  the  subject  was  left  to  start  the  first  scenario  message  on 
his  own.  He  was  instructed  to  contact  the  experimenter  via  a phone 
located  in  his  cubicle  if  he  had  additional  questions. 

The  scenario  prepared  by  CDCINCSG  contained  49  messages  for  each 
subject  to  process.  Each  message  was  typed  out  automatically  by  a 
computer-driven  typewriter.  After  a message  was  typed  out,  the  subject 
tore  it  off,  called  up  the  appropriate  format  via  his  CRT  and  began  to 
format  the  contents  of  the  message.  Of  the  49  messages,  J8  required  one 
format  to  complete,  7 required  two  formats,  and  4 required  three  formats 
for  a total  of  64  formats.  When  he  had  completed  all  formats  required 
by  a message  and  entered  the  information  into  the  computer  data  base, 
the  next  message  began  to  type  out. 

Ten  different  format  types  were  used  in  the  experiment.  The 
frequency  of  use  of  each  type  is  shown  in  Table  1. 


Table  1 

FREQUENCY  OF  USE  OF  EACH  FORMAT 


Name  Frequency 

Intelligence  (l  Format) 

Enemy  Situation  Data  Message  J6 

Friendly  Operations  ($  Formats) 

Front  Line  Trace  Data  Message  7 

Command  Post  and  Center  of  Mass  Data  Message  3 

General  Outpost  and  Combat  Outpost  Data  Message  8 

Operational  Area  Data  Message  2 

Phase  Lines  Data  Message  2 

Axis  of  Advance  and  Direction  of  Attack  2 

Objectives  Data  Message  2 

Movement  Control  Points  Data  Message  1 

Fire  Control  Lines  Data  Message  1 

64 
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The  format  selection  process  was  not  investigated  In  the  present 
experiment,  as  this  process  has  been  examined  in  an  earlier  ARI  research 
project.8 

The  experimental  conditions  are  shown  in  Figure  1.  In  each  of  the 
first  four  conditions,  automatic  tabbing  was  provided.  That  is,  after 
completing  each  data  entry  item  the  subject  was  instructed  to  press  the 
"send"  button  (transmit  key)  which  automatically  moved  the  cursor  to  the 
next  entry  item,  eliminating  the  need  to  space  over  from  one  item  to  the 
next.  In  condition  I,  the  subject  called  up  a full  format  such  as  the 
one  shown  in  Figure  2 with  the  cursor  (solid  line)  positioned  under  the 
first  space  of  an  entry  item.  A second  CRT  (CAMl),  positioned  beside 
the  first,  displayed  Instructions  and  tables  of  legal  entries,  where 
appropriate,  for  the  particular  item  the  subject  was  completing.  For 
example,  the  first  CAMI  display  for  the  sample  format  in  Figure  2 is 
shown  in  Figure  3a*  After  the  subject  entered  a message  number  and 
pressed  the  send  button,  the  cursor  on  the  input  CRT  would  move  to  the 
next  entry  (precedence  = PREC  [ ])  and  the  display  shown  in  5b  would 
appear  on  the  CAMI  CRT.  The  subject  would  continue  to  complete  data 
entry  items  (skipping  over  items  for  which  the  message  contained  no  data) 
in  this  manner  until  he  reached  the  end  of  the  format. 

In  condition  II  (see  Figure  l),  only  the  input  CRT  was  used.  To 
complete  the  format,  subjects  in  this  condition  referred  to  instructions 
and  tables  of  legal  entries  contained  in  a manila  folder  which  served  as 
a handbook.  The  handbook  also  contained  a sample  message  worked  out  for 
each  format  type.  In  all  other  respects,  condition  II  was  similar  to 
condition  I. 

Condition  III  contained  an  extra  step.  When  the  subject  called  for 
a format,  Instead  of  receiving  the  full  format  such  as  shown  in  Figure  2, 
he  received  a checklist  of  format  items  such  as  shown  in  Figure  4.  He 
proceeded  to  type  an  "X"  to  the  left  of  each  entry  for  which  the  message 
contained  data.  After  selecting  the  required  items,  he  pressed  the  "send" 
button  and  received  a format  which  consisted  of  only  mandatory  entries 
and  those  items  he  selected.  For  example,  if  the  message  contained  no 
data  for  "RESTR"  or  any  item  below  " -CO-ON- LINE- POSIT"  in  Figure  4,  he 
would  leave  these  entries  blank  and  put  an  X beside  the  others.  Upon 
pressing  the  send  button,  he  would  see  the  display  shown  in  Figure  5. 

For  each  item  selected  by  the  subject  in  Condition  III,  he  received 
corresponding  instructions  and  legal  entries  tables  on  a CAMI  CRT. 
Condition  IV  followed  the  same  checklist  procedure,  but  the  subjects  did 
not  have  the  CAMI  CRT  and  were  required  to  consult  their  handbook.!  for 
instructions  and  tables  of  legal  entries. 


8 Baker  et  al.,  I969,  op.  cit. 
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Instructions,  codes,  legal  entries 
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Figure  1.  Experimental  conditions 


cursor 


UC1  FLT  COORD  DATA 


MSG-NR[!  ]PREC[  ]SCTY[  ]ORIG[  ]RESTR(  ) 
UNIT-ID-OR-TF-NAME[ 

F LX- COORD- 1 ( 

FLT -COORD- 3 ( 

FLT-TIME( 

CO-ON-LINE- POSIT  ( / 

BN-ON- LINE- POSIT ( / 

LT-COORD-PT( 

LT-COORD-PT-TIME( 

RT- COORD- FT  ( 

RT-COORD-PT-TIME( 


Figure  2.  Sample  of  full  format. 


UC1  FRONT  LINE  TRACE  DATA  MESSAGE 

THIS  MESSAGE  ALLOWS  THE  COMMANDER  TO  DEFINE  THE 
POSITION  OF  HIS  TROOPS  ON  THE  GROUND. 

MSG-NR:  THIS  FIVE- CHARACTER  FIELD  PROVIDES  AN 

INDEX  NUMBER  FOR  THE  MESSAGE.  YOU  WILL  HAVE  TO 
NUMBER  EACH  MESSAGE  CONSECUTIVELY,  STARTING  AT 
3a  45601.  THIS  IS  A REQUIRED  ENTRY. 


UC1  FRONT  LINE  TRACE  DATA  MESSAGE 

PREC:  THIS  IS  THE  PRECEDENCE  OF  THE  MESSAGE. 

YOU  WILL  SEND  ALL  MESSAGES  WITH  A PRECEDENCE  OF 
[R]  (ROUTINE)  UNLESS  OTHERWISE  SPECIFIED.  THIS 
IS  A REQUIRED  ENTRY. 

MEANING  LEGAL  ENTRY 


)FLT- COORD-2  ( ) 

)FLT- COORD-4  ( ) 

) 

) 

) 

) 

) 

) 


ROUTINE  R 

PRIORITY  P 

IMMEDIATE  I 

3b  FLASH  F 


Figure  3.  Sample  of  first  two  CAMI  displays  for  format  shown  in  Figure  2. 
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UC1  FLT  COORD  DATA  LIST 


t 

r 

i 

| 

* 

!: 


- RESTR 

- UNIT- ID-OR-TF-NAME 

- FLT-COORD-1 

- FLT-COORD-2 

- FLT-COORD-5 

- FLT -COORD- 4 

- FLT-TIME 

- CO-ON- LINE- POSIT 

- BN-ON- LINE- POSIT 

- LT- COORD- FT 

- LT- COORD- FT -TIME 

- RT-COORD-FT 

- RT-COORD- IT-TIME 


t 

i 


Figure  4.  Sample  of  format  checklist. 


UC1  FLT  COORD  DATA 

msg-nr[  ]prec[  ]scty[  ]orig[ 

UNIT- ID-OR-TF-NAME[ 

FLT-COORD-1 ( ) FLT- COORD-2 ( 

FLT-COORD-3 ( )FLT-COORD-4( 

FLT-TIME ( ) 

CO-ON-LINE-POSIT  ( / ) 


] 

] 

) 

) 


Figure  5.  Sample  of  format  after  subject  has  completed  checklist. 


In  Condlticn  V,  the  subject  was  provided  with  a single  blank-screen 
CRT  and  a handbook  containing  the  Instructions  and  legal  entries.  He 
was  required  to  type  In  both  the  data  entry  Item  name  and  the  appropriate 
data. 


Independent  Variables 

The  experiment  was  conducted  as  a 2x2  factorial  with  two  methods  of 
format,  and  two  levels  of  CAMI  (Table  2).  The  Blank  Format  condition 
(V)  was  compared  with  the  other  handbook  conditions  (II,  IV)  in  a 
separate  analysis.  As  shown  in  Table  2,  12  subjects  were  randomly 
assigned  to  each  experimental  condition. 
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Table  2 


EXPERIMENTAL  CONDITIONS 


Format 

Input  Assistance 

Method 

CAMI  CRT  Handbook 

(I) 

(II) 

Full 

S,-S,„ 

s,,-s„ 

1 12 

13  24 

(III) 

(IV) 

List 

S25‘S56 

S37'S48 

(V) 

Blank 

— 

S49'S60 

Now,  S * Subject. 


Dependent  Variables 

The  dependent  variables  were  accuracy  and  speed.  Accuracy  was 
measured  as  the  percent  correct  of  the  total  number  of  format  entries. 
Thus,  for  each  formatted  message,  an  accuracy  score  was  computed  as: 
Accuracy  * Number  of  correct  data  entries /Total  number  of  entries.  Speed 
was  initially  measured  from  the  time  of  completion  of  one  format  to  the 
time  of  completion  of  the  next.  Later  the  computer  program  was  revised 
to  separate  the  time  required  for  the  message  to  be  typed,  the  time  taken 
by  the  subject  to  select  a format,  and  the  time  consumed  in  composing 
the  format.  Speed  could  then  be  measured  as  elapsed  time  between  the 
selection  of  a message  format  and  the  completion  of  that  format.  Thus, 
for  the  checklist  conditions  (ill  and  IV)  the  elapsed  time  included  the 
time  required  to  place  an  X beside  the  items  to  be  completed. 

Because  this  programming  revision  occurred  after  ten  subjects  had 
completed  the  experiment  (2  per  condition),  the  sample  size  for  time 
scores  was  reduced  from  12  to  10  per  condition. 

The  experiment  was  concerned  with  obtaining  an  estimate  of  accuracy 
and  speed  of  performance  after  the  basic  task  was  well  understood  by 
subjects.  Therefore,  the  first  seven  formats  completed  by  each  subject 
were  considered  practice  exercises  and  were  not  included  in  the  analyses. 
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RESULTS  AND  DISCUSSION 


The  present  experiment  was  designed  to  investigate  the  overall 
effects  of  CAMI  and  formatting  method  on  the  accuracy  and  speed  of  message 
inputting.  A median  accuracy  and  a median  format-completion  time  was 
obtained  for  each  subject.  The  average  median  accuracy  for  each  group 
of  12  subjects  is  shown  in  Figure  6.  Two  analyses  of  variance  were 
conducted  on  these  scores;  the  first  considered  the  first  four  conditions 
in  a 2x2  design  (i-IV  design)  and  the  second  compared  condition  V with 
condition  II  and  condition  IV  (II,  IV,  V design).  All  effects  in  both 
analyses  were  nonsignificant.  The  average  median  format-completion 
time  for  each  group  of  1C  subjects  is  shown  in  Figure  7*  Two  analyses 
of  variance  similar  in  design  to  those  for  the  accuracy  scores  were 
conducted  on  these  completion-time  scores.  The  I-IV  design  analysis 
yielded  no  significant  differences.  The  II,  IV,  V design  resulted  in  an 
F(2,27  df)  of  3*84,  significant  at  the  .05  level.  A Newman-Keuls  sequen- 
tial range  test  identified  the  source  of  this  effect  as  a significant 
difference  (p  < .05)  between  the  average  median  format-completion  times 
of  condition  II  and  condition  V,  a difference  of  4.6  minutes. 

The  data  were  characterised  by  a high  degree  of  variability  (Table 
3),  which  obscured  any  real  experimental  effects  which  may  have  been 
operating.  Thus  the  difference  of  nearly  two  minutes  per  format  between 
the  full  format  (l , II ) and  the  checklist  (III,IV)  conditions  was  insuf- 
ficient, in  the  light  of  score  variability,  to  result  in  a significant  F 
in  the  analysis  of  variance. 


Friendly  vs.  Enemy  Messages 

The  scenario  package  used  in  the  present  experiment  contained  one 
enemy  message  format  and  nine  friendly  message  formats.  Also,  friendly 
and  enemy  messages  were  not  equally  distributed  throughout  the  scenario. 
Thus,  differences  in  performance  comparisons  between  friendly  and  enemy 
format  completion  accuracy  and  speed  may  have  been  due  to  a peculiarity 
of  the  single  enemy  format  used  or  to  the  uneven  distribution  of  friendly/ 
enemy  messages.  However,  for  baseline  purposes  and  because  there  is 
little  other  data  on  the  question,  performance  differences  between 
friendly  and  enemy  inputting  accuracy  and  speed  will  be  reported. 

For  all  conditions  of  the  experiment,  friendly  formats  were  completed 
with  greater  accuracy  and  speed  than  enemy  formats.  The  enemy  messages 
were  intuitively  felt  to  be  more  difficult  to  format  than  the  friendly 
messages,  requiring  more  analytic  and  subjective  judgments  on  the  part 
of  the  subject.  An  analysis  of  variance  on  data  from  the  four  main 
experimental  conditions  (i-IV)  revealed  that  the  mean  accuracy  of  76.9$ 
for  friendly  formats  was  significantly  greater  than  the  mean  accuracy 
of  69.6$  for  enemy  messages,  F(l,44  df)  * 20. 54,  p < .001.  The  mean 
inputting  speed  of  7.1  minutes  for  conditions  I-IV  was  significantly 
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Table  3 


MEDIAN  FORMAT - COMMIT I ON  TIME  IN  SECONDS 
FOR  EACH  SUBJECT  IN  EACH  CONDITION 


Experimental  Condition 

Subjects  Within  Condition 

I 

11  III  IV 

V 

1 

2 65 

33C 

329 

475 

455 

2 

34b 

392 

358 

484 

511 

3 

408 

429 

376 

490 

635 

4 

410 

439 

454 

506 

663 

3 

427 

516 

464 

522 

6&9 

6 

442 

524 

496 

531 

729 

7 

467 

555 

517 

532 

776 

3 

509 

555 

518 

558 

887 

9 

520 

701 

775 

883 

1187 

10 

1023 

764 

1769 

1150 

1383 

faster  than  the  11,8  minutes  per  format  time  required  to  input  enemy 
messages,  F(l,36)  = 104. 37,  p < .001.  Thus  in  the  present  experiment  the 
enemy  messages  were  more  difficult  to  input  than  friendly  messages;  the 
enemy  formats  were  less  accurate  and  required  more  time  to  complete. 
However,  the  design  of  the  experiment  was  more  concerned  with  providing 
a realistic  scenario  to  subjects  than  with  controlling  for  number  and 
distribution  of  friendly  and  enemy  messages.  Thus  the  generalizability 
of  friendly/enemy  differences  found  here  is  questionable. 


Probability  of  Undetected  Error 

Overall  accuracy  scores  represented  the  sums  of  each  of  six  different 
error  types  identified  in  the  present  research.  The  six  types  of  errors 
and  their  definitions  are: 

1.  Commission  error--subject  enters  data  in  a space  which  should 
have  been  left  blank. 

2.  Omission  error — subject  leaves  a space  blank  when  it  should  have 
been  filled. 
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3.  Glossary  error--subject  enters  data  in  the  correct  space  which 
are  incorrect  for  the  entry  at  hand.  For  example,  selection  of  an  in- 
correct entry  from  a legal  entry  table  would  result  in  a glossary  error. 

4.  Category  error — subject  enters  data  which  belong  in  a different 
item  entry  space.  Thus,  if  a subject  enters  a unit  location  in  a date- 
time space  he  conmits  a category  error. 

5.  Abbreviation  error — subject  incorrectly  abbreviates  an  entry. 

'.'cry  often  it  is  difficult  to  decide  whether  a subject  has  made  an 
abbreviation  error  or  a typographic  error.  For  example,  if  a subject 
enters  '*TNKS"  and  the  correct  answer  is  "TANKS"  he  may  have  made  an 
abbreviation  error  or  simply  forgotten  to  type  an  "A"  after  "T".  Thus 
as  an  operational  definition  an  abbreviation  error  was  defined  as  a 
typographic  error  which  otcurs  more  than  once. 

6.  Typographic  error — subject  types  an  incorrect  character,  omits  a 
character,  or  spaces  incorrectly  while  completing  an  entry. 

Categorising  the  different  types  of  errors  permits  identification  of 
which  types  of  errors  would  be  detected  and  which  would  not  be  detected 
by  an  automated  edit  and  validation  routine  such  as  the  one  planned  for 
TOS^.  A known  ratio  of  undetected  to  detected  errors  is  useful  for  real- 
world  information  systems,  for  which  answer  sheets  are  not  usually  avail- 
able. If  valid,  it  would  allow  one  to  estimate  on  the  basis  of  detected 
errors  the  numbers  of  errors  which  entered  the  system  undetected. 

Of  the  types  of  error  listed  above,  commission,  omission,  and 
glossary  errors  will  be  undetectable  by  computer  error  checking.  One 
exception  is  the  case  of  mandatory  entries  in  which  the  computer  would 
detect  omission  errors.  However,  in  this  experiment  as  well  as  an 
earlier  experiment,®  mandatory  entries  were  seldom  omitted. 

Table  4 shows  the  error  rates  for  each  error  type,  both  detectable 
and  undetectable,  under  friendly  and  enemy  conditions.  A very  striking 
point  is  the  fact  that  most  of  the  errors  identified  in  this  experiment 
would  not  have  been  caught  by  a computerized  edit  and  validation  routine. 
For  both  friendly  and  enemy  conditions  the  ratio  of  undetected  to  detected 
errors  is  more  than  four  to  one  l 

The  high  ratio  of  computer-undetected  to  detected  errors  in  the  pres- 
ent research  suggests  that  users  should  be  trained  to  a high  level  of 
proficiency  prior  to  using  the  system.  However,  when  such  systems  are 
fielded  and  are  operating  in  a combat  environment  it  may  not  be  possible 
to  have  trained  personnel  on  hand  for  replacements.  One  solution  to  this 
difficulty  would  be  to  incorporate  an  automated  traii.ing  package  into  the 


9 Strub,  1971,  op.  clt. 
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Table  4 


MEAN  ERROR  RATES  FOR  EACH  ERROR  TYPE 
ACROSS  FRIENDLY  AND  ENEMY  MESSAGES 


Friendly 

Enemy 

Error  Type 

Error  Rate  $ Total 

Error  Rate  $ Total 

Not  detectable 

Commission 

.064 

Omission 

.053 

Glossary 

.069 

.186 

Detectable 

Category 

.018 

Abbreviation 

.001 

Typographic 

.026 

.045 

26$ 

.018 

6$ 

23$ 

.148 

48$ 

30$ 

.087 

_22L 

81$ 

.253 

83$ 

8$ 

.023 

8$ 

0$ 

.013 

4$ 

11$ 

.016 

19$ 

.052 

17$ 

system.  Such  a package  would  allow  the  user  to  sit  down  at  the  very 
device  he  will  be  operating  and  receive  automated  instruction  on  the  task 
of  transforming  and  inputting  tactical  data. 

What  is  the  difference  between  the  automated  training  program  advoca- 
ted above  and  the  CAMI  procedure  explored  in  the  present  experiment?  The 
key  difference  is  that  CAMI  was  designed  to  function  operationally  in  a 
real  message  inputting  situation.  An  automated  training  program,  on  the 
other  hand,  would  make  maximal  use  of  feedback  or  knowledge  of  results 
in  the  course  of  bringing  a user  to  a desired  level  of  profioiency  prior 
to  assigning  him  duty  as  a regular  operator. 

Automated  training  such  as  this  poses  a real  challenge  in  terms  of 
discovering  an  optimal  strategy  for  carrying  out  the  instruction. 

Atkinson  and  Paulson,10  in  discussing  the  psychology  of  instruction,  point 
out  the  capability  of  the  computer  for  administering  parameter-dependent 


10Atkinson,  R.  C.  and  J.  A.  Paulson.  An  approach  to  the  psychology  of 
instruction.  Psychological  Bulletin.  1972,  j8,  49-61. 
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optimization  strategies.  This  is  possible  because  of  the  ability  of  the 
computer  to  be  sensitive  to  each  individual's  response  and  sets  of 
responses.  Parameter-dependent  models  are  potentially  valuable  because 
they  take  into  account  individual  differences  among  learners. 


SUMMARY 

In  summary  then,  it  may  be  stated  that  the  CAMI  aid  had  no  significant 
effect  on  the  time  required  to  transform  and  input  free-text  messages  into 
the  computer's  data  base.  The  checklist  method  also  had  no  significant 
effect  on  message  processing  time  over  the  full-format  method.  The 
blank-screen  condition,  however,  required  a significantly  greater  amount 
of  time  to  process  messages.  The  CAMI  aid  and  checklist  method  had  no 
apparent  effect  on  accuracy. 

Messages  requiring  the  enemy  format  were  more  difficult  and  required 
more  time  to  process  than  messages  requiring  friendly  formats,  but  for 
reasons  discussed  earlier  this  finding  should  not  be  generalized. 

There  appears  to  be  a great  danger  that  most  errors  in  on-line  data 
inputting  systems  may  not  be  detectable  by  computerized  error- checking 
routines.  Over  80$  of  the  errors  in  the  present  experiment  would  not 
have  been  caught  by  such  a routine.  An  automated  training  package  to  be 
inserted  into  the  system  might  reduce  errors  by  providing  needed  instruc- 
tion to  less  experienced  operators. 


CONCLUSIONS 

The  primary  objective  of  the  present  experiment  was  to  evaluate  (l) 
a computer-assisted  message  Inputting  (CAMl)  aid  which  automatically 
provided  to  the  input  operator  information  useful  for  completing  the 
format,  and  (2)  a checklist  method  of  formatting  free-text  information 
into  computer-acceptable  terminology.  The  dependent  variables  were  the 
accuracy  and  the  speed  of  message  inputting  as  measured  by  median 
accuracy  and  median  format-completion  times.  Analyses  of  variance  on 
these  scores  revealed  no  significant  effects  attributable  to  CAMI  or  the 
format  checklist  method. 

The  present  experiment,  however,  demonstrated  the  need  for  separating 
overall  error  rates  into  component  categories.  Categorization  of  errors 
makes  it  possible  to  determine  the  probability  that  an  error  would  be 
detected  by  an  automated  error- correct ion  procedure.  Such  a statistic 
could  be  used  to  estimate  the  frequency  with  which  errors  go  undetected 
into  a real-world,  on-line  information  system.  In  the  present  experiment, 
over  80 $ of  the  errors  would  not  have  been  detected  by  a computerized 
edit  and  validation  routine. 
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According  to  this  finding,  computerized  error  checking  may  not  sub- 
stantially compensate  for  lack  of  proficiency  on  the  part  of  the  input 
operator.  However,  in  the  tactical  environment  in  which  this  system 
may  operate,  a large  reserve  pool  of  trained  personnel  may  not  be 
available.  One  response  to  this  contingency  would  be  to  incorporate  an 
on-line  training  program  into  the  system.  Additional  research  is 
necessary  to  determine  the  optimal  strategy  for  carrying  out  such  a 
program  of  instruction. 
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APPENDIX 


PRELIMINARY  EXPERIMENTAL  INSTRUCTIONS 
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PRELIMINARY  EXPERIMENTAL  INSTRUCTIONS 


The  experiment  in  which  you  are  participating  is  designed  to  test 
the  relative  efficiency  of  several  methods  of  entering  data  into  a 
military  computer. 

You  will  be  placed  in  a fictionalized  combat  situation  and  will  be 
asked  to  enter  data  into  a computer  terminal  as  directed  by  your  CO,  or 
his  S2  and  S3-  The  coordinates  you  will  enter  correspond  to  the  mapsheet 
in  front  of  you  and  you  are  encouraged  to  use  it  to  follow  the  progress 
of  the  "battle",  if  you  wish. 

Messages  will  arrive,  one  at  a time,  via  the  typewriter  on  your 
right.  You  are  not  expected  to  be  able  to  complete  all  messages  in  the 
time  allotted  you.  Work  at  your  own  rate  and  remember  that  accuracy  is 
most  important. 

Many  times  you  will  be  given  messages,  all  of  whose  data  elements 
are  not  on  the  list  of  legal  entries.  In  these  cases,  use  your  own 
judgment  to  choose  the  legal  entry  most  closely  approximating  the  word 
in  the  message. 

Once  again,  remember  that  speed  is  not  vital  in  this  experiment,  but 
accuracy  is. 

We  appreciate  your  cooperation. 
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